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SUMMARY 

In integrated impact assessment there are many different ways in which 
different methods can be combined in an investigation. The most cost- 
effective combinations must be assessed in relation to each assessment: the 
purpose of the assessment, the particular issues to be addressed, the 
stakeholders involved and the ways in which findings are to be used. The 
answers to these questions will depend on the purpose of the investigation. 
They will also depend on the particular context of the investigation, and the 
particular dissemination needs of the end-users. 

This paper begins by looking at the methods and key issues in integrated 
assessment. It goes on to examine the different stages of the investigation 
process - the methods, pitfalls and ways forward, and finally reviews 
institutional considerations. 
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SECTION 1 COLLECTING INFORMATION: METHODS AND KEY ISSUES 
IN INTEGRATED ASSESSMENT 

1.1 Collecting information: key steps in investigation and underlying 
assumptions 

All impact assessments involve collecting information. This is done mainly 
through asking questions of various types, but also through observation. 
There are a range of techniques, details of which are given in the papers on 
different methodologies elsewhere on this site and are not repeated here (See 
Box A). 

The process of collecting information however involves much more than listing 
a few questions and then asking them in the field or conducting a few PRA 
exercises. Whatever methods are used in the investigation, a number of 
different steps are involved. All these steps are important for the reliability, 
relevance and cost effectiveness of the investigation. These steps apply to 
both the assessment as a whole and also to each individual part of the 
investigation: 

• Reviewing existing information: Any investigation needs to 
complement rather than duplicate existing information, otherwise it 
wastes the time and resources of both funders and respondents. 

• Establishing and maintaining rapport: No matter how sophisticated 
the questionnaire or sampling design, if the investigation fails to 
establish rapport with communities or particular respondents reliability 
of the responses obtained is likely to be highly questionable. 

• Recording the information: Even where questions are asked and 
reliable responses given, if this is not appropriately recorded then the 
information cannot be meaningfully analysed. 

• Checking for reiiabiiity both within and between interviews is 
necessary to establish the reasons for any contradictory information 
and/or the limitations of any practical conclusions which can be drawn. 

• Ending the investigation: The ways in which both 
interviews/workshops and the assessment as a whole is ended is 
crucial not only for moral reasons, but also for making a useful 
contribution to programme/policy change and to the reliability of any 
subsequent interviews and assessments. 

This paper discusses ways in which different methods can be cost effectively 
integrated at each stage to increase the reliability of information obtained. 

The discussion is based on a number of underlying assumptions about the 
nature and purpose of impact assessment: 
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• The assessment should be able to make realistic recommendations 

for improvement in programmes and policies 

• Any one assessment should as far as possible contribute to setting up 
of sustainable learning systems for programme and policy 
improvement over time 

• Stakeholder participation, particularly of the poor and most 
vulnerable, is essential to both making realistic recommendations and 
improving programmes and policies in the longer term. 

These requirements mean that the assessment can itself be a cost-effective 
contribution to development rather than an additional cost. They do not in any 
case necessarily substantially increase the costs of impact assessment. They 
require rather a different approach which fully integrates concerns of 
programme/policy change, sustainable learning and stakeholder participation 
at all stages of the investigation, rather than treating them as an additional frill 
in the form of one dissemination/policy workshop at the end of the 
assessment. The implications of these assumptions for the investigation 
process are discussed in detail below. 



BOX A:THE INVESTIGATION PROCESS 
UNDERLYING ASSUMPTIONS 

• policy relevance 

• sustainable learning 

• stakeholder participation 

PROCESS 

Step 1 : Reviewing existing information 

Step 2: Establishing and maintaining rapport 

Step 3: Asking the questions 

Step 4: Recording the information received 

Step 5:Checking for reliability 

Step 6: Ending the investigation 

TECHNIQUES 

• structured questionnaires (See EDIAIS document on Quantification 
Methods ) 

• semi structured and informal interviews See EDIAIS document on 
Qualitative Methods ) 

• case studies ( Qualitative Methods) 

• observation, photography and video ( Qualitative Methods ) 

• PLA diagramming tools (See EDIAIS document on Participatory Methods ) 
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1.2 Balancing competing priorities: fundamental questions 

Any investigation will require resources and takes up the time and energy of 
respondents, interviewers and funders. It is therefore important that these 
resources are used cost-effectively and not wasted collecting a lot of 
information which is never used. This requires assessments to be carefully 
focused on particular practical or policy issues. In order to produce realistic 
recommendations for programme and policy improvement it is not however 
sufficient to rely solely on a priori identification of hypotheses which are then 
tested by statistical quantitative methods. As discussed in detail elsewhere 
on this site (!! Insert link to paper on data analysis forthcoming) conventional 
statistical analysis is insufficient even for establishing reasons for any 
correlations identified. In order to produce realistic recommendations the 
assessment needs to: 

• increase understanding of the nature of development processes 

and the complex interactions between programme interventions, 
contextual factors and the aims and aspirations of the intended 
beneficiaries. This requires investigation not only of quantitative 
impacts at beneficiary level, but their attitudes and aspirations. It also 
requires detailed institutional and contextual analysis. 

• explore realistic possibilities for innovation and change. These 
may not be immediately apparent from statistical analysis of impacts 
of existing policies and practises. It is likely to require careful analysis 
of particular cases of success or failure and/or participatory 
consultation regarding what different stakeholders see as possible 
ways forward. 

• as far as possible gain support for implementation of the 

recommendations and identify any potential problems or conflicts of 
interest between stakeholders. 

It is crucial that assessments go beyond mechanical questioning of 
intended beneficiaries to a much broader and in-depth understanding of 
the complex ways in which development interventions contribute to 
processes of change. This involves assessment at a number of different 
levels: 

• intended beneficiaries differentiated by different stakeholder 
categories and including potential beneficiaries, dropouts and those 
excluded from programmes and policies as well as actual beneficiaries 

• institutional analysis not only of implementing programmes and their 
interventions but also other institutions involved eg government 
agencies, other NGOs operating in the area, grassroots organizations 
and traditional power structures 
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• contextual analysis of both micro and macro-level opportunities and 
constraints affecting both intended beneficiaries and involved 
institutions 

Understanding these different levels is important not only as a focus of any 
investigation which aims to produce realistic practical recommendations, 
but to the design of the investigation process itself. 

Achieving the difficult balance between potentially competing priorities of: 

• careful focus to maintain rigour on the one hand and 

• breadth and depth of investigation to maintain credibility of practical 
conclusions on the other within 

• given resource, time and contextual constraints 

involves asking a number of fundamental questions as outlined in Box 2. 
It is only when these issues have been thought through that the details of 
design of the investigation can be considered - although it is likely that 
these questions will also need to be continually revisited as new possible 
uses for the information present themselves and/or unexpected challenges 
arise. 

There are no blueprints or ' definitive questionnaires ' for any topic, but 
only a broad framework of questions which must be adapted in relation to 
the particular purpose of any one assessment. Even within a broad remit 
of policy relevance, assessments may have a variety of purposes. 
Commonly different stakeholders have different agendas which need to be 
acknowledged and addressed. These will affect the main focus of the 
broad questions to be asked, the degree of detail which is needed in 
relation to each and the ways in which different stakeholders need to be 
involved. Much of the discussion in this paper uses as illustration the 
issues involved in two rather different examples: 



Illustrative examples of requirements of different types of assessment 

1) Programme improvement: to find out how particular loan conditions in a 
microfinance programme should be improved. The intention is to introduce 
any required changes quickly in response to high drop out rates. Here a 
sophisticated statistical questionnaire about impact on incomes for a large 
random sample which takes a long time to analyse is likely to be of limited 
use. More useful would be a series of carefully planned focus group 
discussions followed by a short questionnaire containing a number of policy 
options canvassed for particular clients to gain consensus about change 
and/or identify differences between different categories of client. 

2) Policy development: to find out the gender impact of particular changes in 
regulatory frameworks for enterprise. The intention is to develop gender 
sensitive policy guidelines and develop structures and networks which take 
into account the needs of different stakeholders, linkages between different 
types of enterprise and particularly focusing on the needs of poor women. 
Here a combination of statistical survey, qualitative research and participatory 
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methods will be needed in order to get beyond simplistic and anecdotal 
information. It will require in-depth analysis of complex informal and maybe 
illegal informal activities and intra-household relations. It will also require 
ongoing analysis of findings and participatory discussions with different 
stakeholders in order to investigate appropriate policy responses by different 
agencies. 

See Appendix for more details. 



Whatever its purpose, any one assessment cannot be seen as an 
independent self-contained event, but the mid stage of a learning process to 
which it can contribute. In order to do this it must: 

• build on what is already known 

• consider how the findings are to be used and by whom 

Few assessments exist in a vacuum. They are generally a follow-up of 
previous studies of varying degrees of reliability. They also are able to draw 
on a wealth of unpublished ' grey material ' and/or undocumented knowledge 
about institutions, contexts and impacts. Much of this information can be 
rapidly collated through use of qualitative interviews and/ participatory 
methods/expert workshops. This enables more cost-effective targeting of any 
one assessment and better design of the investigation. In some cases the 
aims of the assessment are to establish a baseline for future impact 
monitoring, or indicators for integration into Management Information 
Systems. 

The final end use of the findings and the disciplinary skills, time and likely 
degrees of interest of the intended end-users are particularly important. There 
is little point in spending scarce resources on a very detailed statistical 
investigation of a particular dimension of impact if the intended end-users are 
unlikely to be able to understand or have the time to read the findings. In this 
case carefully selected case studies together with participatory quantification 
of a few straightforward questions may be sufficient. This does not mean a 
sacrifice of credibility and rigour, but a much tighter focusing of the questions 
to be asked and careful attention to their presentation and analysis on the 
limitations on conclusions which can be drawn. Alternatively other 
stakeholders like local academics or higher level policy-makers must be 
identified who are capable of using the information and conveying it to 
practitioners. On the other hand if the assessment is intended to be a critical 
contribution to both academic and policy debate, then statistical rigour and the 
rigour of qualitative investigation will be very important. Again there must be 
careful attention to presentation of both complex statistical argument and 
qualitative and participatory information. In most cases, a compromise 
between these two extremes is likely to be needed which will both: 

• yield certain sorts of information easily and quickly and 

• require other information in more depth and with greater rigour but with 
a less tight time schedule. 
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The institutional context is important not only for the reliability of findings 
and recommendations, but again for the design of the investigation. Where 
programmes are operating well and have reliable information systems it may 
be possible to rapidly collect reliable and detailed information building on 
existing information. For most purposes it is also likely that programme staff 
can play a significant role in collecting further information. This improved staff 
understanding will then improve programme implementation. Where on the 
other hand investigation involves programmes which are functioning badly, 
policy areas and regulation which are widely resented and evaded and/or 
where corruption is rife collecting information will need to be done by 
independent researchers. Contexts like emergencies and countries with 
political instability and oppression are also likely to present problems. 

The ways in which an appropriate balance can be found will depend partly on 
the socio-economic and institutional context. Understanding of context is 
not only essential for the reliability of findings and recommendations, but also 
design of the investigation itself. Some questions are easier to investigate in 
some contexts than others. Which questions are likely to be sensitive and 
which can be rapidly and reliably assessed cannot necessarily be foreseen. 
Assessing levels of income may be very sensitive where there are high levels 
of culturally unacceptable or illegal activities like prostitution or cross-border 
smuggling. Asking questions about intra-household decision-making on the 
other hand may be less sensitive than often assumed provided questionnaires 
are based on prior qualitative or participatory research to identify context- 
relevant questions and how they should be asked. 



Examples of unexpected problems 

In South Africa Small Enterprise Foundation found that questions about asset 
values were extremely sensitive as people were unwilling to disclose second- 
hand values of property. This made any quantification of impact on incomes 
extremely problematic (personal communication from Anton Simanowitz). 

In Cameroon attempts to get participatory indicators for women's poverty and 
assess income levels proved extremely problematic. Firstly women asserted 
that they were all poor and were very reluctant to give any relative ranking. 
The main criteria referred to were marital status and age with junior wives in 
polygamous marriages, widows and elderly women being generally agreed to 
be the poorest. Secondly women frequently worked many different plots of 
varying distances and productivity under a range of tenure agreements under 
customary arrangements and with landlords and often did not know the area 
of each plot. Produce was sold or consumed as circumstances required and 
no records were kept. Most women traders and entrepreneurs were also 
farmers. On the other hand attempts to develop participatory indicators for 
empowerment proved far easier focusing on women's control over their own 
income and data on this was far easier to collect (Mayoux 2001 ). 
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More detailed discussion of issues involved in selecting indicators and 
samples is given elsewhere on the website. Indicator trees can be used to 
identify priority areas of investigation and sections of a questionnaire and 
indicate interrelationships between them (See EDIAIS document on Selecting 
Indicators ). Stakeholder analysis should be used to identify samples for both 
quantitative and qualitative investigation, stakeholders to participate or be 
represented in participatory exercises and workshops and end-users of the 
information. Stakeholder analysis should clarify ways in which the 
investigation can be made relevant to the various stakeholders, including the 
poorest and most disadvantaged. It should also clarify the ways in which 
methodologies and questions might need to be modified for different groups 
e.g. for women and men or different ethnic groups, and also the ways in which 
dissemination might need to take into account their different levels of access 
to particular media, different skills and understandings (See EDIAIS document 
on Stakeholder Analysis and document on Sampling ). 

The interpersonal dimension of the investigation are likely to be critical to 
the investigation process: who is to be involved at different stages and how 
they are likely to relate to different stakeholders. Even the best and most 
sophisticated questionnaire and sample design will not give reliable or useful 
information unless the investigation is done well and carefully by sincere 
investigators who have good interpersonal skills. It is not necessarily the case 
that information is best collected by external researchers, in some cases it 
may be done better by programme staff, grassroots groups or even illiterate 
women themselves given appropriate support. Each assessment must 
examine the various types of information needed and the most appropriate 
type of investigator on its own merit. In some cases sensitivity of context or 
topic may necessitate particular recruitment requirements eg gender and/or 
ethnic background, particular linguistic skills and/or educational/disciplinary 
background. In other cases it may be appropriate to maintain an equal 
opportunities policy for recruitment of investigators as a moral issue in itself 
and/or in order to challenge particular gender, ethnic or other stereotypes. In 
some contexts local village researchers or grassroots group leaders may be 
better able to get reliable information than educated outsiders from urban 
areas who do not speak the local dialect and/or are perceived as 
unsympathetic to local customs. In other contexts local researchers and 
leaders may be perceived as too partial or not to be trusted and outsiders may 
be able to get more reliable information, partly because of their novelty value 
and higher levels of confidentiality. These are issues which must be at least 
considered in the interpretation of the information collected, even if for various 
reasons they cannot be foreseen or addressed in investigator recruitment. 

Finally there are a number of cross-cutting ethical issues which must be 
considered at all stages of the investigation. This is not only a moral issue but 
also has implications for the reliability of data collected. Certain types of 
information, for example particular sources of income, intra- household 
relations or levels of corruption within communities or programmes may be 
very sensitive. At the same time in some contexts widespread occurrence of 
black-market activities, secret saving and/or bribery make any estimate of 
income which ignores these activities will be very unreliable. Investigation of 
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levels of intra-household conflict and violence may be essential to 
understanding women's strategies and high incidence of female and child 
poverty. In many contexts these are important factors in explaining issues like 
women's savings and loan use, enterprise behaviour and patterns of income 
earning. These issues are discussed in more detail in some of the Case 
Studies on this site (Insert links to Self-Help Development Foundation Case 
Study). Where it is necessary to collect such sensitive information it is crucial 
that the confidentiality of information from particularly vulnerable respondents 
is observed in order to protect them from repercussions within households, 
from other community members and/or programme staff or other 
administrative authorities. 



BOX 1: COLLECTING INFORMATION: FUNDAMENTAL QUESTIONS 



FUNDAMENTAL QUESTIONS 

• What is the purpose of the assessment? How does it fit into ongoing 
learning processes within programmes and institutions? Whose agendas 
are involved? What does this imply for stakeholder representation at 
different stages of the investigation? 

• How is the information to be used and by whom? What does it imply 
for the speed and timing of dissemination of findings? For priorities of the 
investigation? For sensitivity of particular findings? For the ways in which 
the findings are to be presented? 

• Are there any particular contextual opportunities and constraints? 

Are there any social, economic or political factors which might make any 
parts of the investigation particularly fruitful or particularly sensitive? Is the 
organizational and institutional context conducive to the investigation or 
likely to be problematic? 

LEVELS OF INVESTIGATION 

• Intended beneficiaries: including observed impacts, attitudes and 
aspirations 

• Institutional analysis: including intended aims of the intervention, 
institutions involved, implementation processes, opportunities and 
constraints of institutional cultures 

• Contextual analysis: including opportunities and constraints of macro- 
economic and political context, ethnic, class and gender relations, local 
market and environmental conditions 

DESIGN OF THE INVESTIGATION 
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• What do we want to know? What does this imply for the main focus of 
the investigation? What does this imply for selection of indicators? The 
level of detail required on different issues? 

• Whom do we ask? What is this imply for the types of questions which 
can be asked? The timing of the investigation? 

• What methods are to be used? What are the potential advantages and 
disadvantages of each? How are they to be overcome through integrating 
methods, crosschecking and triangulation? 

• Who is to be involved in the investigation? Which parts of the 
investigation are best conducted by external researchers, programme 
staff, grassroots groups? Are there any special requirements for particular 
questions or respondents in terms of gender, ethnic background, 
educational/ discipline/skill background, language proficiency and so on? 

CROSS-CUTTING ETHICAL QUESTIONS 

• Is the collection of sensitive material absolutely necessary? How is it to be 
used? 

• How are vulnerable respondents to be protected? 

• How are positive findings to be used? 

• How are negative findings to be dealt with? 



1.4 Integrated assessment: building on complementarities between 
methods 

Many impact assessments have mainly used simple survey questionnaires 
with relatively small samples of people. Others have used a limited number of 
PLA techniques and focus group discussions. Yet others have used informal 
interviews with a small number of respondents. All of these approaches 
based on a single methodology have yielded information of limited value in 
terms of its reliability, its representativeness and/or its usefulness for 
programme improvement. 

Integrated assessment by contrast seeks to combine different methods at 
different stages of the investigation in order to build on their 
complementarities and address their weaknesses. A summary of the different 
methods, their requirements, advantages and pitfalls based on discussion 
elsewhere on the site is given in Box 2 (See documents on Quantitative 
Methods, Qualitative Methods and Participatory Methods ). Integrating 
methods is not just a question of using different methods at different stages of 
an investigation, although this is useful. Methods can also be integrated at 
each stage, for example: 
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• Qualitative unstructured interviews on attitudes or more sensitive 
issues may be added on to a structured questionnaire either as follow- 
up to particular questions or as a separate section. 

• Questions on attitudes and perceptions indicated through qualitative 
methods can be added to surveys and questionnaires for 
quantification. 

• Short quantitative surveys are useful as an introduction to qualitative 
interviews and criteria for deciding which respondents should be 
investigated in most detail can be specified in advance or developed as 
the investigation progresses. People are often used to surveys. In 
some contexts this may be a less public way of beginning an 
investigation of sensitive issues than participatory methods. They may 
also be a good way of introducing more qualitative interviewing and/or 
ensuring that certain quantitative information on respondent 
background and certain key indicators exists for analysis of qualitative 
information. 

• Focus group discussions may be used to bring people together and 
then individuals surveyed or interviewed privately in a separate room 
while others attend the focus group. This is often a time-effective way 
of integrating methods where populations are very dispersed. 

• Observation techniques including photography and video can be used 
throughout all types of investigation to give an idea of the reliability of 
information obtained, possible other important areas of investigation 
and possible limitations to the generalisability of information from 
particular respondents. 

These various ways of integrating methodologies at different stages of the 
investigation are discussed in more detail below. 



BOX 2: COLLECT 


NG INFORMATION 


: INTEGRATING M 


ETHODS 




Statistical 
methods 


Qualitative 
methods 


Participatory 
methods 


Types of 
questions 


• What is 
happening? 

• Who is it 
happening to? 

• Why do people 
think it is 
happening? 


• What is 
happening? 

• Who is it 
happening to? 

• Why is it 
happening? 


• What is 
happening? 

• Who is it 
happening to? 

• Why is it 
happening? 

• What can be 
done about it? 


Techniques 
used 


• Structured 
questionnaires 


• Semi- 
structured and 
informal 
Interviews 


• PLA 

diagramming 
tools 
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• Case studies 

• Observation, 
photography 
and video 




Scope of 
indicators 


• Narrow range 
of indicators 
which can be 
measured or 
enumerated 


• Broad range of 
complex 
indicators 


• Broad range of 
measurable 
and complex 
locally- 
identified 
indicators 


Sampling 
methods 


• Large random 
samples 


• Small 
purposive 
sample 


• Combination of 
small and large 
purposive 
samples 
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Main method of 


• Statistical 


• Qualitative 


• Diagrams with 


analysis and 


analysis with 


information, 


textual 


dissemination 


tables and 


generally in 


information 




charts 


textual form but 


• Participatory 




• Report/s for 


sometimes as 


workshops 




selected policy- 


diagrams 


which give 




makers at 


• Report/s for 


immediate 




specific stages 


selected policy- 


information to 




of the 


makers at 


participants at 




investigation 


specific stages 


all levels 






of the 


• Participatory 






investigation 


dissemination 








and policy 








workshops for 








selected 








stakeholders 








both grassroots 








and policy 








makers 


Stages of 


• To investigate 


• Exploratory 


• Establishing 


investigation 


the 


investigation of 


rapport with 


where most 


generalisability 


context and 


communities 


useful 


of findings from 


meanings 


and individuals 




qualitative and 


which will be 


• Exploratory 




PLA methods 


attributed to 


investigation of 




• May be good 


particular 


context and 




introduction for 


questions 


meanings 




use of other 


• In-depth 


which will be 




methods and/or 


investigation of 


attributed to 




identifying 


particularly 


particular 




samples to be 


sensitive and 


questions 




investigated in 


complex issues 


• Investigation of 




more detail 


• Follow-up on 


possible policy 






correlations 


responses to 






identified by 


findings 






quantitative 








methods or 








findings of PLA 




Advantages 


Potential 


• Can get good 


Rapid collection 




elimination of 


understanding 


of data on broad 




sample bias 


of complex 


range of issues 






issues 


Potential to 






• Can get 


directly combine 






information on 


investigation with 






sensitive 


programme 






issues 


improvement 


Potential pitfalls 


• Narrowness of 


• Potential for 


Potential bias in 




analysis 


bias in samples 


both responses 




• Potential 




and samples 
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superficiality or 
falsification of 
responses to 
long or difficult 
questionnaires 







SECTION 2: DIFFERENT STAGES OF THE INVESTIGATION PROCESS: 
METHODS, PITFALLS AND WAYS FORWARD 

2.1 Reviewing existing information 

Failure to build on existing information wastes the time and resources of 
funders. It is likely that those interviewed will also rapidly become tired of 
answering similar questions in different assessments unless the usefulness of 
doing so is clearly apparent to them. This will seriously affect the reliability of 
any information obtained. 

Reviewing existing information includes looking at obvious sources like 
programme records and statistics. Even where previous impact assessments 
have not been conducted, reviewing existing information is essential for 
institutional analysis 1 which is in turn necessary for explaining outcomes and 
identifying ways in which different institutions are to be involved in the 
assessment itself. There is also generally a wealth of contextual information 
from local research institutions, government offices, NGOs and other local 
organisations. These can reveal information on health, incidence of domestic 
disputes, political participation of women, ethnic minorities and so on. 
Participatory workshops and focus groups are also often useful for identifying 
a range of sources of information, identifying gaps and shortcomings and 
establishing rapport as discussed below. 



Examples of use of existing information 

Microfinance programmes often have comprehensive statistical information 
on background of clients, repayment patterns and activities for which loans 
were said to be taken. Although this last cannot necessarily be taken at face 
value because it is often unreliable, this information could be analysed in 
much more detail to identify ways in which services could be improved. For 
example seasonal differences in repayment patterns for clients from particular 
backgrounds can identify probable incidence of seasonality of certain 
dimensions of poverty. The statistics can often be analysed in much more 
detail to reveal gender differences in programme participation, loan taking and 
savings deposits, loan sizes and extent of repeat loans which can indicate 
some of the gender equity issues to be addressed and/or the different needs 
of women from different economic, ethnic or marital status. Similar analysis is 



1 For a very useful discussion of using existing information for institutional analysis see 
O'Laughlin 1998 for analysis of 'grey literature'; Roche 1998 for more general institutional 
analysis and analysing accounts Jones 1998. For a discussion of gender-sensitive 
institutional analysis see Goetz 1992, 1995. 
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usually possible for ethnic differences, rural/urban differences and so on. This 
analysis can be a good way of identifying particular priority questions, 
samples for surveys, client categories for whom different loan packages may 
be needed and preliminary indication of any seasonal dimensions or 
requirements of particular economic activities. 

For policy research there are often a range of existing sources of information 
from research institutions, NGOs and government offices. Participatory 
workshops with different carefully selected stakeholder groups is also likely to 
be a very useful way of rapidly collecting preliminary information. Participants 
can be asked to bring along any existing information they have, or even do 
some preliminary documentation of their experience or investigation of 
existing knowledge of groups with which they work. 



Once the range of existing information has been assessed, its reliability and 
coverage should be reviewed: 

• whether or not the information can be used directly in the current 
assessment, 

• whether or not it can be built on through using supplementary 
questions 

• or whether it is too limited, unreliable or out of date to be used at all. 

This includes for example examination of the frameworks used and whether 
or not they are adequate to answer questions involved in the current 
assessment, which types of respondents have been covered and whether or 
not any relevant groups or geographical locations have been excluded, 
whether or not the information is out of date. 

Even where existing information cannot be used, it is useful to be able to 
explain why this is the case and the rationale for the current assessment to 
both respondents and collaborating institutions. Where it can be used it can 
greatly increase the cost effectiveness and reliability of an impact assessment 
through for example: 

• providing baseline information on at least certain respondents who can 
be revisited, 

• providing statistical information against which the likely generalisability 
of in-depth qualitative research can be at least roughly assessed or 

• highlighting key questions to be asked and some of the pitfalls to be 
avoided in asking or phrasing questions. 

2.2: Establishing and maintaining rapport 

Establishing and maintaining rapport both with communities and stakeholders 
for the investigation as a whole and also at the beginning of each individual 
interview or participatory exercise is crucial. Any investigation takes up the 
time of respondents and/or intrudes on their privacy. In some cases the 
investigation may be a welcome diversion in a dreary daily routine. 
Respondents may readily see the importance of the investigation and be 
grateful for an opportunity to voice their opinion. In many cases however 
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respondents will be extremely busy and have little time to spare. Although 
they may see the point of certain questions and be eager to discuss particular 
issues, they may be less enthusiastic about other questions, particularly those 
which are collecting detailed information about their personal lives and which 
seem to have little benefit for them. Unless rapport is established the answers 
they give are unlikely to be reliable. 

Establishing and maintaining rapport with different institutional interests is also 
critical to the outcomes of the assessment. Bad relations with powerful 
interests can jeopardise the whole assessment and/or prevent collaboration 
by vulnerable groups. Good relations are essential to ensuring realistic 
practical recommendations which have been broadly discussed to test their 
acceptability and possible limitations. Good relations are also essential if the 
recommendations made are to be broadly supported and implemented. 

As noted above, familiarity with any other studies or research on similar 
issues is very useful introducing the assessment and explaining its 
significance. It is important to clarify the expectations which different 
stakeholders might have of the assessment itself and potential outcomes. In 
some cases the assessment may be linked directly to changes in 
programmes and policies, as in the microfinance example above. In others, 
as in the gender policy example above, the outcomes may be less direct or 
certain. Currently powerful stakeholders may even be disadvantaged relative 
to certain currently disadvantaged stakeholders. 

Deciding how the linkages between the assessment and outcomes for 
participants should be explained is obviously tricky. Unless handled 
sensitively, respondents may falsify information depending on what they 
expect the outcomes to be. It is therefore crucial that if direct linkages 
between the assessment and outcomes is to be made, that the importance of 
obtaining accurate information is stressed and the commitment to balancing 
competing and potentially conflicting interests in the interests of those 
disadvantaged is emphasised and maintained. Where no direct outcome is 
likely, this should be made clear to respondents from the outset and their 
participation in the investigation must be encouraged through other means. 
Where very detailed information is needed on particular topics, the reasons 
for such detail must be explained to respondents and the interview made as 
much fun or as useful to them as possible. Where good relations are 
established it is likely that substantial amounts of useful and reliable 
information can be collected by grassroots groups and local people if they are 
appropriately supported. 



Examples of ways in which investigation can be made useful for 
respondents 

• detailed questions about income can be presented in ways which help 
people understand the relationship between their cash and income flows 
and the reasons for any problems. 



Howdowefindout-collectinginformation Linda Mayoux 



17 



• detailed questions about decision-making in the household and women's 
status can be conducted in ways which both encouraged women to think 
about gender inequality and ways which they can address them. Such 
questions addressed to men can also perform an awareness-raising 
function. 



The best methods to be used for establishing rapport will depend on the 
context, the sensitivity of questions to be asked and particular stakeholders 
involved. In cases where the issues are relatively uncontroversial, where 
favourable institutional context exists and stakeholders are likely to be 
articulate then participatory methods may be the most cost-effective means of 
rapidly introducing the investigation to a large number of people and building 
support. Where the issues are much more sensitive and/or the institutional 
context is difficult and/or respondents unused to participating in public fora 
then participants observation, informal interviews or initial surveys containing 
uncontroversial questions may be the best means of introduction and 
ensuring participation of particularly disadvantaged people. It is likely that 
different methods for establishing rapport will be needed for different 
stakeholder groups and possibly for different stages of the investigation if the 
issues to be addressed become progressively more sensitive. 

Whatever methods are used, establishing rapport requires good interpersonal 
skills on the part of the investigating agencies and also an understanding of 
context. It requires: 

• treating all respondents and their needs with respect and appreciating 
their generosity in giving their time for the assessment 

• sensitivity to the respondents' mood, body language and time 
constraints and to the different cultural norms that may shape these; 

• making interviews and discussions fun through using humour and 
personal experience to lighten up long list of questions, bring up 
sensitive issues or to challenge a response 

• respect for any fears or reservations which they may have about 
potential outcomes from the investigation. 

These points apply to each individual interview or participatory exercise even 
where rapport has been established at the community level. 

2.3: Asking the questions 

As noted above, there are no blueprints or ' definitive questionnaires ' for 
any topic, but only a broad framework of questions which must be adapted in 
relation to the purpose of an investigation, how the findings are to be used 
and by whom. As noted above indicator trees can be used to identify 
particular areas of question and the particular methods to be used for each. 
Stakeholder analysis can be used to identify whether different methods are 
likely to be needed for particular questions for different stakeholders. 
Examples of frameworks of questions for particular topics of investigation in 
different areas of enterprise development are given elsewhere in the web site. 
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Detailed questionnaires used in particular studies can also be found in some 
of the resources mentioned at the end of these papers 2 . What follows here 
are some general practical considerations in using and integrating the 
different methods following our discussion above and the questions in Box 2 
(See Box 3). 

If the assessment is to yield useful policy recommendations then policy- 
related questions based on institutional analysis need to be integrated into 
questionnaires, interviews and participatory exercises where possible. This 
includes not only analysis of reasons why particular impacts are occurring, but 
attitudes and aspirations of the different stakeholders involved and also their 
views on possibilities for change. Questionnaires, interviews and participatory 
workshops can also be used to identify ways in which ongoing structures for 
programme or grassroots level learning could be set up and maintained. This 
must however be done sensitively, bearing in mind institutional constraints 
and the probability that even with the best will in the world it will not be 
possible to fulfil all the demands made. The possibility that expectations of 
programme change may influence responses given must also be borne in 
mind. It is best therefore if policy-related questions are left to the end of a 
questionnaire and/or crosschecking questions are included. Alternatively, 
where the situation is very sensitive, policy-related questions could be 
canvassed for a carefully selected sample of key stakeholders after analysis 
of the key findings as a follow-on to this. 

Throughout the process of asking questions, interpersonal issues are 
extremely important, not only for ethical reasons but also to increase the 
reliability of information 3 . It is important that time and resource constraints of 
respondents also be taken into account in the design of questionnaires, 
interviews and discussions. This is as true of 'grassroots' research like diaries 
and so on as of more conventional investigation by outsiders. As noted above 
there are ways of making questionnaires useful for respondents and the 
timing and location of interviews and participatory workshops should be 
convenient for participants to ensure a relaxed and open atmosphere. It is 
important for the interviewer or facilitator to be self-critical of their own value 
judgements and avoid leading questions. They should also observe 
behaviour as well as recording verbal responses. 

In the case of statistical surveys where many people are to be interviewed 
and data needs to be quantified for statistical analysis, sessions and 
questions should be kept to a minimum and as simple and short as possible. 
The degree of precision of quantification should be clear, but there should 
always be space for recording qualitative information where necessary. 



2 Detailed questionnaires for microfinance can be found in The SEEP Network 2000. Detailed 
frameworks used for gender and empowerment are given in the TSP paper on women's 
empowerment. Detailed frameworks of questions used in Fair Trade are given in the Case 
Studies of Oxfam Fair Trade and Just-X. Further outlines for constructing questionnaires are 
given in Nichols 1990 and Roche 1999. 

3 A stark example of the importance of interpersonal relations is given in the SHDF 
microfinance case study (Insert link forthcoming) where there was a very considerable 
discrepancy in answers to even straightforward questions like loan use depending on the 
relationship between interviewer and despondent. This is by no means an isolated finding. 
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Questions should be free of value judgements and not biased towards 
particular responses. Questions should ideally be asked in the same way from 
all respondents. Otherwise differences in responses may be due to 
differences in the ways in which questions asked rather than actual 
differences between respondents. This is a very tricky issue. Not all 
respondents will understand the same question or words in the same way and 
it may be differences in understanding rather than real differences which may 
cause different responses. A balance must therefore be drawn between 
standardisation on the one hand and flexibility to different potential 
respondent understandings on the other. Where either standardisation or 
flexibility in questions is not possible the possible implications must be borne 
in mind at the analysis stage. 

Use of qualitative methods is very different. Informal interviews and case 
studies need to be flexible to the priorities and interests of respondents as 
well as those of interviewers. Although there must be a rough outline of the 
questions to be asked, these need not be asked in the same order for each 
respondent or focus on the same issues. Questions should be open-ended in 
order to invite detailed responses. Again attention needs to be paid to the 
ways in which the data are to be analysed. Particularly in short investigations, 
it is often possible to visualise the way in which an interview or case study will 
be used or written up and this helps guide the line of inquiry. Direct 
observation needs to balance specificity and reliability of evidence observed 
with openness and flexibility to the unexpected. 

Detailed guidelines for participatory workshops are given elsewhere in this 
web site and in numerous manuals on the topic (See Participatory Methods ). 
It is possible to use participatory methods to get quantitative as well as 
qualitative information. It is important that participatory exercises have a clear 
focus and build on each other to maximise complementarities between 
different types of participatory tool. Otherwise there is a danger that the 
investigation becomes vague and/or mechanical. It is also important to 
ensure that all participants are able to contribute and that power relations and 
inequalities do not bias the outcomes. This can be done through: 

• the ways in which particular techniques are used and the ways in 
which responses are probed 

• careful selection of participants in each workshop and/or separating 
stakeholder groups within particular workshops 



BOX 3 PRACTICAL CONSIDERATIONS IN INTERVIEW/WORKSHOP 
DESIGN AND PROCESS 

GENERAL QUESTIONS 

• What do we want to know? What does this imply for selection of 
indicators? For background information on each participant? For 
general contextual information? For information on organisational and 
institutional context? 
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• Whom do we ask? What does this imply for the types of questions 
which can be asked? The timing of the investigation? The types of 
methods to be used? 

• How is the information to be analysed and disseminated? What 
does this imply for the levels of reliability required? For the use of 
quantification? For the use of illustrative case studies? For the design of 
participatory workshops? 

• What does this imply for methods to be used? For indicators and 
background information to be quantified? For areas to be covered by 
qualitative informal interviews and observation? For issues to be 
covered in participatory exercise is? 

GENERAL POINTS APPLICABLE TO ALL METHODS 

• Where possible include questions and/or discussion of reasons why 
particular impacts are occurring and exploration of possible solutions 

• Make the interview as interesting and useful for respondents as 
possible 

• Timing and location should be responsive to the needs of respondents 
and the possibility for open and frank discussion 

• Avoid leading questions and value judgements 

• Observe the behaviour of respondents and those about them 

GUIDELINES FOR STATISTICAL SURVEYS 

• Each session should be as short as possible without unduly 
compromising reliability and depth 

• Each question should be distinct and not conflate different issues. 

• The degree of precision of quantification required should be clear with 
separate space left for recording qualitative information. 

• Questions should be as clear, short and specific as possible inviting a 
limited range of responses which can be quantified. 

• Question should be value-free and not biased towards particular 
responses 

• Words should be simple, direct and familiar to all respondents and 
should not be value-laden. 

• Questions should be applicable to all respondents using the same 
language and asked in the same way. Where this is not possible 
implications must be borne in mind at the analysis stage. 

GUIDELINES FOR INFORMAL INTERVIEWS AND CASE STUDIES 

• Questions or topics should be tailored to different informants and 
stages of enquiry making use of findings from previous interviews 

• Open-ended questions should be used to follow the line of the 
interview as it evolves with the respondent and probe more deeply 

• Informants can be identified progressively to explore a range of 
different types of knowledge and perspectives 

GUIDELINES FOR DIRECT OBSERVATION 
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• The particular behaviour activities or issues to be observed need to be 
identified 

• There need to be clear guidelines and explanations of how evidence is 
to be assessed and what types of behaviour are to taken as evidence 

• There needs to be an assessment of the representativeness of the 
particular context observed 

• There needs to be provision for the unexpected 

PARTICIPATORY EXERCISES AND WORKSHOPS 

• Each participatory exercise should have specified objectives 

• Exercises should complement and build on rather than repeat each 
other 

• Everyone should be encouraged and able to participate equally 



2.4: Recording the information received 

The range of information which could be recorded is almost infinite. Very 
simple questions in surveys may not accurately reflect the complex 
reality of particular respondents and respondents may give long and 
complex answers. Much of this information will be relevant to interpreting 
the answer which is finally recorded on the piece of paper and to 
explaining patterns or abnormalities identified in statistical analysis. 
Similarly many contextual factors affect the responses given and are 
relevant to their interpretation. Unfortunately the relevance of many 
details may only become evident in the light of subsequent investigation 
with the same or different respondents. Which particular details are 
relevant cannot therefore always be predicted with any certainty, 
particularly at the beginning of an investigation. At the same time it is 
obviously not possible to put down everything. Recording information 
may disrupt the flow of communication, may seem threatening to those 
interviewed and/or may increase the time needed in analysis. 

There is therefore difficult balance to be struck between comprehensive 
recording on the one hand and manageability of information on the other. 
This is always tricky and with hindsight decisions may not always have 
been correct. Nevertheless a number of general guidelines can be 
applied as indicated in Box 4. As far as possible the exact words of an 
interviewee should be recorded and separated from remarks and 
interpretation of the interviewer. Note taking should be as inobtrusive as 
possible. Particularly for the qualitative dimensions of an investigation 
and/or participatory workshops tape recording or videoing may be the 
least obtrusive method. It ensures that that all information is retained for 
crosschecking if need be and only the points which are judged to be 
most important that the time need be recorded during the interview or 
workshop. Photographs may be extremely useful even in surveys for 
corroborating certain evidence on e.g. general indicators of poverty like 
housing, health status and so on and for recording particular types of 
contextual information. 



Howdowefindout-collectinginformation Linda Mayoux 



22 



Certain minimum details of the context of any particular interview or 
participatory exercise should be noted as this is likely to influence the 
reliability of the data obtained. Even in statistical surveys this may 
explain certain patterns or anomalies identified. For example a particular 
batch of respondents may give particular answers because they have 
been asked in a different location or different season from others. 
Individual respondents may give apparently contradictory or nonsensical 
answers because other particular individuals were present which meant 
they falsified their answers because of fear of repercussions. 

In statistical surveys it is important to both standardise the ways in which 
responses are recorded and quantified and to note any qualitative 
information which may affect responses. It is crucial that there is 
consistency between interviewers as well as interviews. This is likely to 
require a pilot phase to identify any inconsistencies in the ways in which 
questions are asked and recorded. It is also important to record certain 
minimum information on the context of each interview. For example if 
questions concern sensitive issues like incomes, intra household 
relations or behaviour of programme staff, it may be relevant to give the 
list of people present when the questions were asked. Time of year or 
location may be relevant. In integrated impact assessment it is important 
to maximise the opportunities offered by the survey interaction to 
observe differences between respondents over a large sample and 
follow-up qualitative areas of investigation as they arise. 

In qualitative investigation the balance between detail and manageability 
of information is particularly difficult. Qualitative methods are used to 
record complexities of situations and processes. It is particularly difficult 
to predict what is likely to be relevant and not relevant. It is therefore 
important to make recording information as true as possible to the 
interviewee's exact words and also to record any information considered 
relevant to its interpretation. In integrated assessment it is also 
important that qualitative information can be used to shed light on the 
findings of quantitative investigation. It is therefore important that certain 
minimal responses are recorded as quantitative measures for example 
on interviewee background and certain key questions. This enables the 
generalisability of particular interviews to be assessed, and also for 
qualitative responses to be compared with the same findings from 
quantitative surveys. 

There are some ways in which the recording of qualitative interviews can 
be made easier through using diagrams to record information, drawing 
on participatory techniques. Tape recording should be done wherever 
possible so that there is a record of the interview as a whole just in case 
certain information needs to be crosschecked at a later date and/or to 
follow-up particular issues which did not appear important at the time. 
Photography and video can be useful in recording particular impacts, for 
example achievements of poor households (See photograph 1), impact 
on gender relations (See photographs 2 and 3). These records must 
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however be accompanied by the necessary contextual information to 
interpret them. 

In participatory exercises the process of participation and discussion is 
important, not only the final diagram product. It is therefore important to 
document this process in detail to identify the ways in which particular 
information is obtained, how consensus is reached and the power 
relations which may have affected this. This involves not only recording 
discussion in the exercises themselves but also detailed observation of 
body language, surroundings and recording details of informal 
interactions. Where possible video should be used to decrease the 
amount of information which needs to be recorded during the exercise 
and free the facilitator and rapporteur to observe the process rather than 
having their head down in note taking. Again this also ensures that a 
record is kept for future crosschecking or following up of issues which 
may not at the time have seemed important. Videoing is however 
inevitably selective and it is important therefore that considerable care is 
taken to ensure that the video is as objective and complete as possible 
rather than attempting to be a selective and interpretive work of art. 
Editing for dissemination can be done at a later stage. 



BOX 4: GUIDELINES FOR RECORDING INFORMATION 
GENERAL 

• as far as possible quote an interviewee's exact words 

• make clear where the interviewer's own analysis and interpretation 
has been added 

• make note taking as inobtrusive as possible. 

• video, photographs and tape recording may be very useful where 
possible 

CONTEXTUAL INFORMATION TO BE RECORDED 

• Who was present? (numbers, relation to respondent, gender, poverty 
status, ethnicity, generation, names where appropriate) 

• Who did the investigation? How might interpersonal factors affect the 
reliability and scope of the information collected? For example linguistic 
problems, gender and ethnic issues. 

• Are any particular characteristics of the location significant? e.g. lack of 
privacy, excessive noise disturbing responses and so on, inability to be 
relaxed? 

• Are any particular characteristics of the timing or length of the interview 
significant? e.g. the respondents being very busy or tired, just having 
had a particular shock or seasonality factors which may influence the 
ways in which incomes and expenditures are perceived and reported? 

POINTS SPECIFIC TO STATISTICAL SURVEYS 

• standardise as far as possible the ways in which responses are 
interpreted and recorded. Where several interviewers are used then a 
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trial set of questionnaires should be conducted and cross checked to 
ensure consistency between interviewers. 

• qualitative notes and qualifications should be recorded, particularly 
where these are relevant to the reliability of the qualitative information 
collected and/or to identify areas to be followed up with qualitative 
investigation 

POINTS SPECIFIC TO QUALITATIVE INVESTIGATION 

• collect detailed contextual information 

• record the interviewee's exact words and also general observations of 
the interviewer 

• responses certain key questions should be recorded systematically to 
allow for quantification at the analysis stage 

• diagrams should be used where possible to reduce the burden of note 
taking and also to facilitate crosschecking of information with 
respondents 

• use tape recording where possible for crosschecking later or following 
up on particular issues 

• use photography and video to highlight particular impacts together with 
the necessary contextual information 

POINTS SPECIFIC TO PARTICIPATORY EXERCISES 

Make detailed notes on: 

• who was present and how they participated, who was not present or 
did not take part in discussions and why. 

• any points of interpretation essential to understanding each visual 
output? e.g. who contributed, degrees of consensus, whether the 
diagram was jointly produced or the product of combining a number of 
separate diagrams, points of disagreement 

• the way in which different individuals and groups treat each other, and 
each other's ideas, particularly the ways in which conflict or 
disagreement between individuals and groups is handled and the degree 
of independent decision-making by different people and groups; 

• the body-language of participants and the physical setup of the house 
or meetings and gatherings; 

• the informal interactions before, during, and after discussions, 
meetings and during breaks. 

• use video where possible for crosschecking later or following up on 
particular issues 



2.5:Checking for reliability 

Ensuring the reliability of particular interviews depends on the levels of 
rapport which have been established and the way in which the 
investigation has been explained to respondent/s. It may be necessary 
periodically to emphasise the importance of giving honest answers where 
there is any doubt. It is also important to address the needs and fears of 
respondents as they arise and to continually be aware of any potential for 
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biases in the interviewer interpretation of responses. Where responses 
are vague or obviously contradictory there are number of probing 
techniques which can be used. Where necessary questions can be 
repeated in a different form provided this is noted so that it does not 
prejudice comparability between interviews. At the end of each interview 
information should be reviewed and read back to the respondent to clarify 
any points which may have been misunderstood or which appear 
contradictory in the light of subsequent information and discussion. 

In integrated assessment information can and should be cross-checked 
through triangulation of methods. At the analysis stage, or even during the 
interview or workshop, it is important to identify which areas of 
investigation required to be crosschecked by other methods. For example 
it may be difficult to explain certain patterns thrown up by quantitative 
methods and these should then be further investigated using qualitative or 
participatory methods. Conversely the generalisability of findings of 
qualitative and participatory methods may need to be crosschecked 
through a survey. It is also important to constantly compare the information 
collected by different methods to see how they are building on each other. 

Crosschecking reliability of information is crucial to ensuring that practical 
recommendations are representative and realistic. Where it has been 
possible to include questions about programme implementation and policy 
in questionnaires and workshops, this can be crosschecked and followed 
up in any workshops at the end of the research. Ideally also any 
recommendations should be discussed informally with the relevant 
stakeholders before presenting to a wider audience. Where the policy and 
institutional context are more sensitive, practical questions about follow-up 
should at least be broached through a participatory workshop which 
produces concrete proposals for follow-up discussions. 



BOX 5: TECHNIQUES FOR CROSS-CHECKING INFORMATION 
GENERAL POINTS 

• continually emphasise where necessary the importance of the investigation 
and of giving honest answers 

• be sensitive to respondents needs and fears and address these wherever 
possible 

• continually examining own biases 

• responses should be read back to the respondent at the end of the interview 

PROBING TECHNIQUES 4 

• Show interest 

• Pause: Silence can tell a respondent you're waiting to hear more 



4 Adapted from Frey and Oishi 1995 'How to Conduct Interviews by Telephone and In Person 
Volume 4 'The Survey Kit' Thousand oaks, CA Sage publications quoted SEEP Network 
2000. 
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• Repeat the question: this can help the respondent who has misunderstood, 
misinterpreted or straight from the question to get back on track 

• Repeat the reply: this can stimulate the respondents say more or to 
recognise the inaccuracy 

• Ask a neutral question: 

• For clarification: ' What you mean exactly? ' ' Could you please explain 
that? ' For specificity: ' could you be more specific about that? Could you 
please explain that? 

• For relevance: ' I see. Well, let me ask you again. ' ' Could you tell me how 
you mean that? ' 

• For completeness: 'what else? Can you think of an example? 



2.6: Ending the investigation 

Ending the investigation properly is extremely important. This is obviously so 
for moral reasons and ethical considerations. It is also crucial to the reliability 
of any subsequent interviews. News of any impropriety on the part of 
interviewers and/or fears of potential repercussions travel fast and may 
seriously jeopardise co-operation in subsequent interviews and/or the 
reliability of responses. The different questions which should be asked at the 
end of each interview or participatory workshop are given in Box 6. 



BOX 6: ENDING THE INVESTIGATION 

• Do respondents wish certain information to be kept confidential? 

• Do respondents have any questions they want to ask the interviewer? 

• Do respondents have any worries arising out of the interview? 

• Which of the findings do respondents feel clearly indicate possibilities for 
improving the intervention/s under investigation? 

• How do they think this can or should be followed up? 

• What does the interviewer honestly feel it is possible to do in response to 
these answers? 



Ideally findings of the investigation and possible policy recommendations 
should be discussed with the relevant stakeholders throughout the research, 
to avoid mistrust and unexpected shocks which might jeopardise future 
relations. This includes both programme staff and other implementing 
organizations as well as intended beneficiaries. Practically, where the 
institutional context is difficult this ideal may not be possible and the ways in 
which findings and recommendations can best be presented to different 
stakeholders may need to be treated with care. It is nevertheless important 
that all these concerns are brought together and addressed at the end of the 
investigation as a whole. This is in order to ensure that the impact 
assessment is integrated into programme and policy improvement. It is also 
important to ensure cooperation in any subsequent investigations. Issues in 
designing participatory dissemination and policy formation workshops are 
discussed elsewhere on the site (Forthcoming document on Information 
Analysis and Dissemination). 
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SECTION 3: INTEGRATED IMPACT ASSESSMENT: SUMMARY 
GUIDELINES FOR COLLECTING INFORMATION 

In integrated impact assessment there are many different ways in which 
different methods can be combined in an investigation. The most cost- 
effective combinations must be assessed in relation to each assessment: the 
purpose of the assessment, the particular issues to be addressed, the 
stakeholders involved in the ways in which findings are to be used. Possible 
ways of integrating methods in relation to the above examples of microfinance 
and gender-sensitive enterprise regulation are given in the Appendix. A 
summary of the general questions to be asked at each stage of the 
investigation based on the above discussion is given in Box 7. 

The answers to these questions will depend on the purpose of the 
investigation as indicated in the examples of microfinance and gender- 
sensitive regulatory frameworks given in the Appendix. They will also depend 
on: 

• the particular context of the investigation, and the suggestions given in 
the Appendix depend critically on the degree to which the assumptions 
indicated are realistic. 

• the particular dissemination needs of the end-users as discussed 
elsewhere on the site (Insert link to 'What do we do with the 
information? Analysis, Dissemination and Implementation final tool 
paper to be produced by end of March) 



BOX 7: SUMMARY QUESTIONS TO BE ASKED AT EACH STAGE 
OF THE INVESTIGATION 

Step 1: Reviewing existing information 

• What information is available and what are its limitations? 

• Which information can be used directly in the proposed assessment? 

• Which information can be built on through using supplementary 
questions? 

• Which information is too limited, unreliable or out of date to be used 
at all? 

• What critical gaps exist which need to be covered to enable useful 
practical recommendations to be made in the proposed assessment? 

Step 2: Establishing and maintaining rapport 

• Who are the key stakeholders? How are they to be involved in the 
research? In particular are programme staff or intended beneficiaries 
to be only respondents, or involved in investigation itself? 

• What are the likely sensitivities, needs and fears to be addressed due 
to the issues to be addressed, the institutional and/or socio- political 
context? 
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• How are these sensitivities, needs and fears likely to differ between 
stakeholder groups? How can any particular sensitivities be 
addressed? 

• How might the methodologies need to be adapted for different issues, 
contexts or respondents? 

• Should or can practical policy questions be introduced at this stage? 

Step 3: Asking the questions 

• How is information on each issue to be collected and by whom? 

• Which particular indicators and variables need to be quantified and 
with what degree of precision? 

• Which processes require investigation by qualitative methods? 

• At which stages are participatory methods needed? 

• What is the best way of integrating practical policy questions? 

Step 4: Recording the information received 

• What is the appropriate balance between simplicity of recording to 
assist analysis and detailed recording to ensure reliability and enable 
crosschecking at a later stage? 

• Is the use of tape recording, photography or video feasible? 

Step 5:Checking for reliability 

• Which of the findings are considered reliable and credible in 
themselves? How do we know? Does the questionnaire or discussion 
have inbuilt crosschecks? 

• What practical policy conclusions are incontrovertible and broadly 
agreed and which are contentious? 

• Which of the findings or policy conclusions require further 
investigation by other methods? Which findings and which methods 
should be used? 

• Are there any relevant aspects of the context which are key to 
understanding the process of investigation? The findings? The policy 
recommendations? e.g. aspects of the social context known to be 
relevant, power relations. Do any of these require further 
investigation? 

Step 6: Ending the investigation 

• How are the findings and practical policy conclusions to be presented 
and to whom? 

• Do respondents have any questions or worries which need to be 
addressed in follow-up? 

• How can the policy recommendations be implemented? How will this 
be followed up? 
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